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ABSTRACT 

Goal five of America 2000 calls for dramatic 
improvement in the educational attainment of the nation's adult 
population. Encompassed in the goal is a charge to Increase the 
number of college graduates who can think c-ltlcally, communicate 
effectively, and solve problems. This paper focuses on measures that 
can be used to assure that this objective is being met. Several 
measures will be needed to address the many aspects of college 
learning that contribute to the development of higher order skills. A 
set of six Indicators is proposed, each of which is expected to 
promote and gauge progress. The proposed measures will foc"s on the 
following areas: (1) opportunities for secondary school students to 
engage in college-level learning; (2) coherence and depth in college 
programs; (3) contributions of disciplinary associations to the 
structure of undergraduate major programs; (4) creation and operation 
of departmental assessment programs; (5) employer and educator 
perceptions of work readiness in college graduates; and (6) levels of 
literacy in adult and young adult populations. Reviews by N. Beck, N. 
Frederlksen, and J. Herman of this position paper are provided. 
(Author/SLD) 
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Abstract 

Goal five of America 2000 caWs for dramatic improvement in 
the educational attainment of the nation's adult population. 
Encompassed within the goal is a charge to increase the 
number of college graduates who can think critically, 
communicate effectively, and solve problems. The Resnick 
and Peterson paper focusses on measures that can be used 
to assure that this objective is being met. 

The authors of the paper believe that a variety of measures 
will be needed to adequately address the many aspects of 
college learning that contribute to the development of higher 
order skills. They propose a set of six indicators—both 
quantitative and qualitative— for this purpose. Each is 
expected to both promote and gauge progress. 

The proposed measures will focus on the following areas: 
opportunities for secondary school students to engage in 
college-level learning; coherence and depth in college 
programs; contributions of disciplinary associations to the 
structure of undergraduate major programs; creation and 
operation of departmental asses^^ment programs; employer 
and educator perceptions of work readiness in college 
graduates; and levels of literacy in the country's adult and 
young adult population. 
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Evaluating Progress Toward Goal Five: 
A Report to ti t National Center for Education Statistics 

Daniel P. Resnick and Natalie L. Peterson 
October?, 1991 



The White House and the governors of the United States are embarked on a course that, if • 
supponed by Congress, will chanss our educational system. Expressing a set of understandings 
reached the previous year at Charlottesville, the administration and the state executives agreed on 
six goals in 1990 that ^ail for a major transformation in the way the American educational system 
operates. Goals three, four, and five call for much higher achievement in the foimal school 
experience of young people and the formal and informal learning of adults. To make the policy 
process responsive to legislators and voters, measures will have to be set up to gauge progress 
toward these goals. Since these learning goals ispresent a bold pohcy agenda to which the 
administration and the governors have committed themselves and which the Congress is being 
asked to shape and endorse, the measures chosen will have a double function. They must not 
only monitor progress toward the goals, but help to make sure that the goals zxt achieved. 

What kind of measures should be chosen? The indicators chosen will not meet oar policy 
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needs if they arc only neutral scientiric measures, though they should be valid and rciiubitj. 
Indicators will be expected to play a role in clarifying the goals and making ihem attainable. 
Because the United States is at some distance from a satisfactor>' approximation of the knov. ledge 
and capabilities that these goals intend for us as a nation — whether it is 'ccmpeier-cy* in 
subject areas or a well-prepared workforce— the indicators will also be expected, perhaps in \ isibic 
ways, to mobilize public action. 

The pressure to mobilize public opinion around these goals comes, in large pan, from the 
American business community, whose leadership has been challenged in N^ orld markets by 
nations with better educated workforces that are exploidng economic opponunides more 
effectively. There is soitjc disagreement in the scholarly literanire about how much education the 
American workplaces of the future will require of their workers.' If v.e extrapolate from curreni 
employment statisdcs, reflecting large numbers of non-skilled jobs that may not survive another 
decade, the demand for a more educated workforce seems very uncenain. If, however, we look at 
the growth rate of new hires in different sectors of the workforce, the sectors with increased 
educadonal demands are certainly growing more rapidly. Leading firms have begun the process 
of re-structuring. They have flattened their organizational hierarchies, broadened the job 
responsibilities of their workers, and provided educational opportunities in the workplace. 

The movement for higher quality and productivity in die woriq)lace has its counteipan in 
the movement for higher standards and restrucmring in public schools. States and districts, 
working with one another, many through the New Standards Project, are making plans to set high 
standards for their pupils and help their young people achieve them. Curriculum, professional 
development and parental support are under development in these programs. The indicators 
chosen to measure progress toward goals three and four will be called upon not only to register the 
variety of changes in learning outcomes that are underway, but also to encourage changes in the 
educational system which will make those outcomes possible. 

' John Bishop, "A Worsening Shonage of College Graduates'* in Educational Evaluation and Policy Analysis. 
(fonhcoming), reviews these arguments. 
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Since the standard of living of the nation depends increasingly on bringing to the United 
States and keeping in this country a large number of highly skilled jobs, the natioRa! policy agenda 
is open for proposals to raise the level of education of school graduates and lo upgrade cap:ibiii'Je:> 
of already eraptoycd workers.' It is within that context of policy action that the goals of 
America's governors and the White House must be placed. The .American political agend;: to 
create a workforce with higher order skills— capable of solving problems, recognizing liie broaaer 
context of woikplace activity, learning how to keep on learning, using technology effectively and 
devising adaptive responses to new challenges. These are challenges that have been set before 
both the population that enters the workplace directly after high school and the segment that enters 
after the compl-ftion of one or more years of college. 

The Colleges and Goal l ive 

A substantial part of the entering workforce each year is composed of those with two or more years 
of college education. The colleges and universities, however, have ?emained largely outside the 
firamework of the movement to raise standards. This is the case even though higher education can 
offer considerable resources to aid in the transfonmation of precoUege learning and can expect 
direct benefits firom die success of this process. As many have noted, leaders in higher education 
seem not to realize the full magnitude of the precollege transformation that is in progress.' In 
some colleges and universities the adaptation to underprepared snidents is complete and the large 
doses of remedial offerings have become an expected feature of the curriculum. This passivity has 
serious implications for the construction of indicators designed to measure transformations in the 
readiness of our young people for working life. 

' National Cenier on Education and the Economy, Commission on the Skills of the American Workforce, 
America's Choice: high skiiis or low wages! (Rochester, NY: National Cenier on Education and the Economy, 
1990) and U.S. Department of Labor, The Secreiary 's Commission on Achieving Necessary Skills, What Work 
Requires of Schools: A SCANS report for America 2000 (Washington D.C. : U.S. Depanmcni of Labor. 1991). 

* Sec for instance. Marc Tucker, "Many U.S. Colleges Arc Really Incfficicnu High-Priced Secondary- Schools" 
in The Chronicle of Higher Education, Volume XXXVll, No. 38, p. A36 and Frank Newman "Resinicturing ihe 
Public Trust" in AAHE Bulletin, September 1990, Vol. 44, No. 1, pp. 3-8. 
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Goal five of America 2000 demands higher qualificaaons from ihe college and non-college 
populations that constitute our workforce: 

By the year 2000, every adult American w\\ be literate and \m*11 
possess the knowledge and skills necessary to compete in a global 
economy and exercise the rights and responsibilities of citizenship. 

Embraced within the goal are the following five objectives: 



• Every major American businesis will be involved with the 
strengthening the connection between education and work. 

• All workers will have the same opponunity to acquire the 
knowledge and skills, from basi: to highly technical, needed to 
adapt to emerging new technologies, work methods, and markets 
tfirough public and private educational, vocational, technical, 
workplace or other programs . 

•The number of quality programs, including those at libraries, that 
are designed to serve more effectively the needs of the growing 
number of pan-time and mid-career students will increase 
substantially. 

• The proportion of those qualified students, especially minorities, 
who enter college, who complete at least two years, and who 
complete their degree programs will increase substantially. 

• The proportion of college graduates who demonstrate an advanced 
ability to think critically, communicate effectively, and solve 
problems, will increase substantially. 

To reach these objectives, a wide-ranging set of indicators will have to be developed. 
They will have the large burden of pointing out the directions in which post- secondary insrinitions 
should shift in order to increase the capabilities of their graduates. The set will have to include 
measures of work readiness, educational equity, program availability, degree completion, and 
college learning. The choice of indicators for this goal is particularly difficult because of the 
scattered age distribution of the working population, the variety of formal and informal means by 
which those in the workforce receive their education, and the many different kinds of demands 
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We have elected to focus on the objective of promoting and measuring college student 
learning, panicularly the learning of four-year college graduates. In our search tor approprlits 
indicators, we will draw on relevant assessment practices in primary' and secondar>' as well as 
higher education, and take account of recent studies on the capabilities of America's young adui: 
population. The indicators that we will propose are intended to press the insntutions, discipiUics, 
students, and teachers functioning together in higher education to change the college learning 
experience so that a larger portion of college graduates can enter the workforce at the higher level 
of attainment which goal five demands. 

In 1988-89, 1,017,667 baccalaureate degrees were conferred at colleges and universities 
across the nation.* Most of the degrees awarded were from large public instinirions varying 
greatly in. quality. Liberal arts colleges represent less than ten percent of the institutions granting 
baccalaureate degrees. In this graduating class, the most popular field of study was business 
administration, where 246,659 degrees were granted. The social sciences ranked second, with 
107,714 degrees conferred. Other leading majors were education, health, and psychology. Any 
effon to raise the exit level skills of these students must recognize that such know-how is subject 
matter dependent and needs to be cultivated through a challenging curriculum that shapes the 
learning of our undergraduates and their major programs. 



What Should College Graduates Know and Be Able to Do? 

What is it that a college graduate should know and be able to do? Though the skills that will be 
demanded by the workplace of the fiihire cannot be predicted with certainty, a number of 
competencies are already recognized as essential. The recent repon by the Department of 
Labor.lVAar Work Requires of Schools: A SCANS Report for America 2000, explores them with 

. ' The Chronicle of Higher Education Almanac. August 28, 1991. Volume XXXVIII, No. 1, p. 3. 
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Employees of the future, the SC iNS repon sigues, will be called upon to demonstrate 
skills in five different Qomains. The first involves the uses of resources. In tomorrow's 
workplace, employees will be repeatedly called upon to schedule time, budget funds, assi^m space, 
and arrange staff. The second domain concerns interpersonal skills. Future workers must be 
adept at working in a team, teaching others new skills, serving clients and custome' s, exercising 
leadership, negotiating, and dealing with diversity. The third area of competency focuses on the 
ability to use and acquire information. Employees will be expected to acquire, evaluate, organize, 
maintain, interpret, and evaluate information. They should also be prepared to use computers to 
process this infonnation. The fourth domain is concerned with systems. Employees must have an 
understanding of how social, organizational, and technological systems work and be able to 
operate effectively with them. Based upon this knowledge, they should be able to monitor and 
correct performance and improve or even design systems. The final area of competency involves 
technology. In the workplace of the future employees will be expected to have a familiarity with a 
variety.of technologies so that they will be able to select, apply, and maintain them. 

Each of these domains is based upon a foundation of basic skills, thinking skills, and 
personal qualities. Basic skills include, at some level, reading, writing, mathematics, listening, 
and speaking. Thinking skills, sometimes referred to as higher order skills, are those associated 
with thinking creatively, making decisions, and solving prob'iems. Related skills include knowing 
how to learn, reason, and organize as well as to process .symbols, pictures, graphs, objects and 
odier infonnation. Personal qualities, described by the report, include responsibility, self-esteem, 
sociability, self management, and integrity. 

An effective approach for promoting the thinking skills and personal qualities identified in 
the SCANS report was adopted by Alvemo College in the early 1970s. The college's curriculum 

U.S. Deparunent of Labor. The Seaetary's Commission on Achieving Necessary Skills, What Work Requires 
of Schools: A SCANS Report for America 2000 (Washington D.C.: U.S. Department of Labor, 1991). 
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and assessment program are desigjied around eight general abilities thai each student is expejied to 
master before graduation. These abilities include: effective communication; analysis; problem 
solving: valuing in a decision-making context; effective social interaction; effectiveness in 
individual/environmental relationships; responsible involvement in the contemporar.- world; and 
aesthetic responsiveness. Students must demonstrate competence at set levels in each of ihc>c 
areas. They are assessed in a variety of settings, using many different modes. ' Several of the 
skills are assessed in public settings, and many are assessed by outside judges v^ iih busineb> and 
ccxmnuniQr experience. 

Clearly, all of the skills and qualities underlying the five domains as well as those within 
the domains themselves are ones that should be possessed by college graduates. Such skills, 
however, will also be needed by those who do not attend college. VVhai then, should dif:erer.i;aie 
college graduates from the rest of the population? Once again, we turn to the notion of demand. 
Society expects college graduates to have accumulated substantial knowledge within a specific 
domain. The math major, for example, should not only be able to perform mathematical 
operations, but should also be able to represent the worlil of experience with the forms, heuristics 
and language of mathematics. The histor>' major should not only be able to construct nairatives 
and place historical figures in their national and social contexts, but should also have a deep 
knowledge of several historical periods including their institutions, cultures, and values. 

The majority of colleges and universities, for historical rtasons, educate Uieir students 
through majon in specific disciplines.. What a college student should know and be able to do 
depends largely upon his or her field of study. The search for students more capable of critical 
thinking will thus depend in some way on the quality of study in the major field Our proposal for 
goal five indicators recognizes this fact. 

ICnowledge of fields outside Uie major field is also essential. The college graduate should 
have a deep understanding of his or her major area of study, but shoula also be able to relate 

• The Alvemo College Facully, Assessment at Alverno College, (Milwaukee: Alvemo Productions. 1979). 

7 

U 



Goal Five Evaluation Resnick & Peterson 

knowledge to other disciplines. This view is well expressed in the Association of Amenc^r, 

Colleges publication, Tlie Challenge of Connected Learning : 

To fulfill its role in liixral learning, the major must also 
structure conversations with the other cultures represented in 
the academy, conversations that more nearly reflect the 
diversities within our world and require patient labors of 
translation.^ 

The problem of translation across fields has grown with knowledge in the disciplines. Each 
student needs experience in both interpreting the meaning of work in other fields and conveying tlie 
meaning of his or her own field to others. The kind of skills and integration that we assumed in 
goal five depend on that training and capability. 



Assessment in Primary and Secondary Education— What Can Be Learned? 



As we seek measures of the knowledge and skills of college graduates, we will want to survey the 
kinds of measures that are used at the primary and secondary level. Are there practices that we 
may want to emulate at the post-secondary level? Have ways been found to gauge student 
learning through ineasurcs that actually promote learning? The answer is yes, for a few select 
programs in which assessment is rich in nature and closely tied to a curriculum. Most of the 
assessment in our primary and secondary schools, however, is not of this kind. 



Hundred of millions of .standardized tests are given annually to the almost fifty million S-to 
17-year-oIds in our schools.' Such tests are firmly entrenched in the American educational 
system, having been used since the Progressive Era for selection, placement, and program 



American Association of Colleges, The Challenge of Connected Learning (Washington D.C.: American 
Association of Colleges, 1991), p. S. 

* Beverly Anderson, "Test Use Today in Elementary and Secondary Schools," in Alexandra K. Wigdor and 
Wendell R.OanKrieds.),AbUityTesdHg: Uses, Consequences, and Controversies. Part II. Committee on Ability 
Testing, Assembly of Behavtonl And Social Sciences, National Research Cbuncil (Washington, D.C.: National 
Academy Press, 1982), pp. 232-85: Lauren B. and Daniel P. Resnick, "Assessing the Hiinking Curriculum: New 
Tcols for Educational Rcfonn,- in B. R. GifTord and M.C. O'Connor (eds.), Changing assessments: Alternative 
Views of Aptittide, Achievement and Instruction. Bosm: Kluwer. 1991, pp. 37-75. 
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evaluation.' Indeed, no other nation relies so heavily on icsts to examine the learning of its young 
people. Yet, as intemational studies of student performance in areas like mathematics achie\ ement 
indicate, our students have scored poorly in comparisons with other countries. In an international 
mathematics assessment conducted between 1980 and 1982, students from a dozen other cour.rrieb 
scored higher than American 13-year-olds in one or more areas of mathematics. Similar results, 
were seen on a 1988 mathematics test where American 13-year-olds had lower scores than their 
peers in four other nations,"* American students arc more accustomed lo test-taking ihdji siUiicnis 
from other countries, but they are not more knowledgeable, 

The majority of tests given in American schools are multiple-choice in format, requiring 
only that students recognize right answers, without the obligation to generate responses 
themselves. While many of these measures are labeled 'achievement' tests, and are expected :o be 
curriculum dependent, they may have little relationship to the intended curriculum of the school 
and classroom. The format of such tests, in any case, serves to decontextualize and decompose 
knowledge." Test takers are not called on to show the capacity for integration and higher order 
thinking that goal five seeks to promote in the workplace. Another problem exists in the fact that 
most of the tests given aire norm-referenced. Norm-referenced measures do not try to gauge the 
student's mastery of a subject area; instead, level of performance is reponed in relationship to a 
distribution of scores. The test-taker need not know very much to perform better than many 
others. Tests of this kind will not show the kinds of growth that we look for in a measure for 
goal five. 

Performance-oriented assessments, based on course material that students have prepared, 
are more likely to be useful for purposes of mobilizing teachers and students to reach the demands 

" Daniel P. Resnick, "History of Educational Testing" in Ability Testing: Uses. Consequences, and 
Controversies. Pan D. Conunitiee on Ability Testing, Assembly of Behavioral And Social Sciences, National 
Research Council (Washington, D.C.: National Academy Press, 1982), pp. 173-195. 

"The National F ucation Goals Panel, The National Education Goals Report (Washington D.C.: National 
Education Goals Panel. 1991). pp. 17-19. 

" Lauren B. Resnick and Daniel P. Resnick, "Assessing the Thinking Curriculum: New Tools for Educational 
Reform." 
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of goal five. Such assessmenis are widely used nationally across a broad sptca^um of course 
programs only in Advanced Placement (AP) and Iniemaiional Baccalaureate (IB) programs. 
Approximately 42 percent of U.S. secondarv' schools, however, offer courses geared toward one 
or more of the of the 28 AP exammations, which have large performance- based componenis.' ' 
The IB program has a small participation rate of only about 150 American schools. Students. \\ ho 
take the AP or IB exams are often awarded advanced standing or college credit based upon their 
performance. Nonetheless, only one in four of the seniors who take the SAT currently register for 
one or more AP courses." Wc believe that indicators should be developed that arc sensitive to the 
availability of the College Board AP program in American high schools, the numbers of students 
who are enrolled, and the success of students on the examinations. Attention to this kind of 
indicator for goal five, will indicate an understanding of the relationship that goal three, four and 
five have to one another and will help school administrators, teachers, parenis, and student.'; to 
understand better the value that this kind of college-level challenge can have in pre-college 
programs. 

The Undergraduate Curriculum: A Tool For Promoting College Learning 

Course requirements arc one of the primary tools that institutions of higher education can use to 
guide the learning process of the college student. To insure that a student has sufficient depth in 
his or her field of study, nx>st colleges require a certain number of courses or credit hours within 
the discipline. For instance, Case Western Reserve University requires its chemistry majors to 
have taken 1 1 courses in the discipline before graduating. Eight or more courses beyond the 
introductory level is a common graduation requirement. 

Many colleges require students to take a specified number of courses outside the major area 
of study. All students enrolled in Florida's system of higher education, for example, are required 

" The College Board, AP Yearbook 1990 (New York: College Entrance Examination Board, 1990). p. 2. 
'* The College Board, College Bound Seniors: 1991 Profile of SAT and Achievement Test Takers (New York: 
College Entrance Examination Board, 1991) and The College Board. A/* Yearbook 1990, p. 4. 
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to take 12 semester hours of coursework in Enclish und six semester hours of mathem;!!;: - 
Other colleges and collegiate structures may tnandate that all students take the same core of courses 
in different disciplines during their freshman and sophomore years. The Universit)' of Chi^zzo 
has a celebrated core program. 

Maintaining an indicator of colleges with distribution requirements, hovxever. •>!..-^.jki 
be a useful way to further our national goals for adult learning. Few colleges, even wiih 
distribution requirements, offer students the opportunity for integrative interdisciplinary study. 
One exception is Miami University, which houses The School of Interdisciplinar>' Studies. 
Students in the program must enroll in a senior workshop where they meet weekly to discux; their 
progress on a required interdisciplinary written project. In the past, students have corr.hined 
disciplines such as chemistry and zoology or music and women's studies." 

Requiring a certain number of courses within specified areas of study does not, however, 
guarantee that a snident will gain the level of understanding that the field demands. There must 
also be some sequence to the courses that the student is required to take. If an English major is 
required to take five courses in the discipline but chooses only those on the introductorj- level, he 
or she would hardly acquire depth of experience and understanding. The Association of American 
Colleges argues that it is important for a major course of stud>' to be based on a principle or 
principles of organization. Reflected within this organization should be "a beginning, a middle, 
and an end— each contribpting in a different but specific way to the overall aim of the major.'"* In 
our view, and we so argue in a later section, an indicator should be developed to express this 
coherence. Instinitions must be encouraged to strive for this goal, and educators and policymakers 
must be provided vvith a way to gauge progress toward it. 

" Terry W. Hanle, "The Growing Interest in Measuring the Educational Achievement of College Siudenis" in 
Clifford Adelman (ed.). Assessment in American Higher Education (Washington D.C.: Depaitmeni of Education, 
1987), p. 5. 

'* Ernest L. Boyer, College: The Undergraduate Experience in America (New York: Harper & Row, 1987), pp. 
259-260. 

'* Association of American Colleges, The Challenge of Connected Learning, pp. 8-9. 
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A study conducted by Robcn Zemsky and his colleagues at the Universii> oi 
Pennsylvania's Institute for Research in Higher Education has made significant headwas in tlub 
direction." Conducted in the late 1980s at the request of the Association of American Colleges" 
Council for Liberal Learning, the study was designed to document the condition of the liberal arts 
curriculum in American colleges and universities and provide some metric by which such 
institutions could gauge their success at providing structure and coherence in their undergraduaie 
programs. 

To gather inforaiation on the undergraduate curriculum, student u-anscripts were collected 
from 30 colleges and universities representing 16 states and every region of the ccuntT>'. Among 
the private institutions were four high-priced selective research universities, five high-priced 
selective liberal arts colleges, six moderately-priced colleges and universities, and six lov.er-pnced 
colleges. Among the public institutions were five large flagship universities with major research 
programs and four predominantly undergraduate teaching instinitions. 

The final transcripts of all students graduating in the spring of 1986 with a baccalaureate 
degree in arts and sciences, business, or engineering were obtained from 28 of these institutions. 
Two others instead submitted transcripts of spring 1987 graduates. Transcripts were supplemented 
by data drawn from course catalogs, and interviews with depanment chairs. A database of the 
collected information was compiled. 

The curricular stnictuic of each of the participating institution was analyzed in terms of 
focus, breadth, and depth. Focus was determined by the extent to which "coursework was 
organized in terms of beginning courses that students would normally take in their first year of 
college, intermediate courses students would take in either their second or third year of college, and 
capstone courses students would normally take in their final year."" Breadth of study was defined 
in terms or the number of courses that students took in math/science, social science, and the 

" Robert Zemsky, Structure and Coherence. 
Ibid, p. 22. 

12 



Goal Five Evaluation Resnick .S: Peterson 

humaniries. Depth was measured by the number of advanced courses — those that cannoi be taken 
by a beginner— that were taken by graduating seniors. This anal) is revealed an undergrad jaie 
curriculum in substantial need of greater structure and coherence. 

Of particular interest were Zcmsky's findings about the depth of the undergraduate 
curriculum." In the study, depth in depanmental registrations in different institutions was 
determined by the portion of courses in the major field taken by seniors which had three or more 
prerequisites. The number of prerequisites was determined by both course catalog review and the 
actual enrollment patterns of students. An area of study was said to have substantial depth if one- 
third of the major cou-ses taken by seniors had three or more prerequisites. If fewer than one in 
six of the courses in the major field taken by juniors and seniors required three or more 
prerequisites, then the area of study was said to be lacking in depth. Using this cnierion, Zemsky 
found that the humanities programs, and their majors, were seriously lacking. Nearly eighty 
percent of the humanities major programs in the sample institutions fell into this categor>'. In only 
ten percent of the sample was there substantial depth af programs. Faring only slightly better, 
were the social sciences, where nearly half of the institutions were categorized as lacking. Only 
mathematics and the natural sciences were deemed to have substantial depth; at least one-third of 
the major programs in those areas at eighty percent of the institutions in the sample had senior 
major courses with three or more prerequisites. 

Guidance is needed from the disciplinaiy associations on how the undergraduate major 

should be constructed. In the American Association of Colleges publication. Reports From the 

Fields, twelve disciplines gave their recommendations. The Mathematical Association of America 

offers the following guidelines: 

Every topic in every course should include an interplay of 
applications, problem solving, and theory.... Every student who 
majors in the mathematical sciences should complete a year-long 
course sequencr. at the upper-division level that builds upon two 
years of lower division mathematics.* 

Ibid, p. 20. 
*• Ibid, p. 78. 
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This course of study should enable the maih major to "undenake iniellectualiy demiir.ding 
mathematical reasoning."" The reluctance of some disciplinary associations to provide such 
guidance may decline if an indicator like the one we propose later in the paper encourages 
professional interest in structuring the undergraduate major. 

It must be remembered, however, that trje depth is a function not onh' of the number of 
prerequisites a course has, and the opponunity that prerequisites provide for students to w ork v\ iih 
advanced students, but of the kinds of educs^tional experiences which the course itself provides. 
Advanced coursework should allow for rich integrative experiences that develop higher order skills 
in disciplinary and interdisciplinary work. Senior year capstone courses, involving 
comprehensive exams, theses or projects, are vehicles for achieving this goal. 

Comprehensive exams during the senior year can require students to pull together and ^ 
integrate what they have learned in their four years of study. Such examinations can take many 
forms. GR£ subject area examinations and professional certification tests are sometimes used for 
this purpose, but the exams may not relate direcdy to the institutions's own curriculum. There is. 
however, increasing use of examinations that are developed by faculty to reflect institutional 
missions and cuiricula. The University of Tennessee at Knoxville, for example, has asked faculty 
in 104 disciplines to agree on performance objectives and comprehensive exams for graduating 
seniors.^ 

A project of the Association of American Colleges, supponed by the Fund for the 
Improvement of Post-Secondary Education, created a network of eighteen institutions to explore 
the effects on student learning and depanmental behavior of comprehensive written and oral 
examination in a half-dozen undergraduate fields. Institutions like Swarthmore College have been 
requiring comprehensive exams, with external examiners, since the 1920s, but the number of 

" Ibid, p. 78. 

" Trudy W. Banta, "The Use of Outcomes Information at the University of Tennessee" in (ed.) Peter T. Ewell, 
Assessing Educational Outcomes (San Francisco: Jossey-Bass Inc., 198S), pp. 19-32. 
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schools with such requirements remains small. The AAC study found that participation in ii'.i.^ 
assessment program did promote student modvation and learning, and that faculty found u a 
rewarding professional experience, but that there were many institudonal disincentives for the 
creation and administration of such programs? Such courses are outside the mold of the rela:i\ ely 
low-cost and low-demand courses that are common fare for senior majors in many disciplines. 

Senior project courses can also function to promote high levels of undersuinding. Tiie\ 
have the potential to strengthen many of the skilh highlighted in the SCANS report. Students may 
be given the opportunity to work in small groups on developing creative solutions to "real world*' 
problems. Students majoring in public policy at Camegie Mellon University, for example, are 
required to enroll in a project course during their senior year. In the past, courses have focused on 
topics such as recycling, hazardous waste, and automobile safety. Students and faculty from 
different disciplines work together to meet the needs of real clients. 

The senior thesis can al.so serve as a valuable capstone experience. In this case, it is not the 
capacity for integration or application that is at issue, but the ability to behave as an expen in a 
defined domain of knowledge. This expenise will be demonstrated in different ways, depending 
on the discipline, but the thesis is both a demanding learning experience and a demonstration of 
what a student knows and can do. Ernest Beyer in College: The Undergraduate Experience, 
recommends that "students be asked to write a senior thesis that would relate the major to 
historical, social, or ethical concerns/* ^ Only a few schools, like Reed College, however, are 
known for the way in which theses shape the senior year experience. 

Assessing Work Readiness Through Employer Surveys 

No set of indicators for goal five can afford to omit the perceptions that employers have of college 

" FIPSE Rqwn, External Examiners and Comprehensive Exams, Association of American Colleges, 
unpublished report The flrst author of this paper served as evaluator. 

** Emesi L. Boyer, College: The Undergraduate Experience in America, p. 259. 
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students as they enter the workforce. An annual employer survey would raise the salience oi" 
employers' expectations for college students at the same time that it would measure progress 
toward greater work readiness. A recent study on the preparedness of high school graduates fcr 
employment and higher education serves as an excellent staning point for developing a suney of 
this kind. Conducted in September 1991 by Louis Harris and Associates, the study was based 
upon telephone interviews with 2,446 employers, educators, parents, recent siudenis. j.'.d 
members of the general public. 

The employers surveyed were 401 line executives and human resource officers randomly 
drawn from Dunn & Bradstreet's listings of businesses and government agencies. Two-thirds of 
those queried were from private industry. Equal numbers of small (S5 to $20 million in sale?), 
medium (S20 to $100 million in sales), and large (over $100 million) companies were represented. 
All other employer respondents were from state and federal government agencies. 

The educator sample was comprised of 301 admissions officers and faculty members from 
equal numbers of trade and vocational schools, two-year colleges, small four-year colleges, and 
large four-year universities. All were selected from Dunn & Bradstreet's listing of educational 
institutions. About half of the questions that educators were asked in a telephone interview, were 
those that employers had also been asked. Also interviewed were 1744 members of the general 
public. From this group, a sample of 250 parents of recent graduates and 511 recently graduat&l 
students was established. Recent graduates were defined as those who had completed high school 
four to dght years ago. 

Telephone interviews for the study were conducted wiih a list of 30 to 70 questions. Each 
major group surveyed was asked a different, but overlapping set of questions. Employers were 
asked to answer approximately 40 questions, mainly about preparedness of recent high school 
graduates for employment within their organizations. Educators were asked 35 questions mainly 
concerning the capabilities of recent high school graduates who had entered their institutions. 
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Parents were asked to assess the abilities of their son or daughter. Forms: students were uike-j \o 
assess their own abilities. Parents and former students were asked appioximately 70 questions. 
All participants were asked a number of questions concerning the quality of primar>' and secondars 
education in America. 

At the core of the employer, educator, parent, and former student protocols \s as a se; of 
items requiring the participant to rate recent hi-^h school graduates on a common set of 15 aitribuies 
related to a young person's ability to perform well in higher education or on the job. Each of the 
new entrants to the workforce, for example, was asked the following question: 

Now, I want you to rate the preparation you feel your high 
school schooling gave you on each of these key areas , On 

do you feel your high school preparation was 

excellent, pretty good, only fair, or poor? 

* 

The survey participants were asked to rate recent high school graduates in each area as excellent, 
pretty good, only fair, poor, or not sure. 

Four of the 15 attributes concerned basic skills in reading, writing, and mathematics. Two 
attributes — the ability to solve complex problems and the ability to read and understand written and 
verbal directions— were higher order in nature. The remaining attributes described personal 
qualities such as having a good attitude and knowing how to dress and behave. The survey 
participants were asked to rate the preparedness of recent high school graduates in each area as 
excellent, pretty good, only fair, poor, or not sure. 

Participant responses concerning the 15 attributes were reported in terms of percent positive 
and percent negative comments. The pattern of responses was clearly displayed, and very 
disturbing for those who seek to advance the agenda of goal five. Employers and educators, on 
one side, and students and parents on the other, have very different and often antithetical views of 
the capabilities of recent entrants into the workforce. Only one-third of the employers surveyed 
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believed ihai recent graduates had the ability to read and understand verbiil and v.riiten insiriicuc^n^. 
Eighty-six percent of parents and recent workforce entrants, by contrast, aftlrmed the abiiit\ of 
recent graduates to and understand oral and written instructions. A gap of approximately 40 points 
divided employers and recent graduates. 

Measuring Literacy-Related Skills 

The NAEP Young Adult Literacy Assessment identified literacy skills as a key element in work 
readiness. Literacy-related skills are a key determinant of an individual's ability to function both 
on the job and in the larger society. These skills include not only reading and writing ai a basic 
level, but also the ability to interpret, extract, and apply information from a variety of texts. The 
NAEP Young Adult Literacy Assessment represents one ver>' sophisticated attempt to gauge such 
skills." Administered in 1985, the survey was designed to measure the level of literacy in 
America's young adult population. We believe that the results of an assessment of this kind could 
serve as a useful indicator for gauging progress toward goal five and displaying literacy 
expeaations for college graduates to a broad public. 

The survey consisted of an oral interview designed to provide background information and 
gauge performance on a series of literacy tasks. Three thousand five hundred young adults 
between the ages of 21 and 25 served as the s mple population. Approximately one-third of the 
interview was devoted to background questions about the participant's activities, occupational 
status, aspirations, education, early language experiences, and household characteristics. This 
information was collected both for descriptive and comparative purposes. Interviews were 
conducted in person by a trained interviewer and lasted approximately 30 minutes. Over 70 
questions were used to elicit the desired background information. 

After completing the background interview, participants were asked to answer nine printed 

Irwin S. Kiisch and Ann Jungblui, Uteracy: Profiles of America's Young Adults (Princeton, New Jersey: 
Educational Testing Service, 1990 (second printing]). Final Report. 
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core questions targeting the most basic literacy skill? Those who were able to ansuer at !tfu>- three 
of these questions were allowed to move on to three sets of 10-15 literacy tasks. A total of 105 
tasks were used in the survey. Respondents unable to correctly answer more than two of the core 
questions were given an oral interview rather than nx>re printed items. Ten oral literacy tasks \\ ere 
administered to this group as well as to a subsample of those who successfully ans\Nered the 
required number of core items. 

Three general types of tasks were used, classified as prose, document, and quantitative. 
Prose tasks required the respondent to demonstrate an understanding of skills associated with 
interpreting and using information from newspaper articles, poems, and other extended textual 
material. Document tasks required the respondent to locate and use information on text? such 
labels, charts, paycheck stubs, deposit slips, and order fomis. Quantitative tasks required 
individuals to perform mathematical operations using figures embedded in a variet)' of text types. . 
A number of questions were also drawn from the 1983-84 NAEP Reading Assessment. These 
multiple-choice items were included so that the performance of young adults in the survey could be 
linked to that of the 9-, 13-and 17-year-olds who participated in the Reading Assessment. 

Tasks were designed to simulate situations in which a young adult would be called upon to 
demonstrate literacy skills. Sometimes, of course, the demands were made for an individual 
response by the reader when the real-world response would have involved conversation with 
others. One task, for example, required respondents to examine an actual bus schedule and 
answer the following question: 



In another, respondents were asked to make distinctions between two types of employee benefits. 
A third involved interpreting instructions from n appliance warranty. Though some tasks, 
presented in written form, required only multiple-choice responses, others were open-ended. 



On Saturday afternoon, if you miss the 2:35 bus leaving 
Hancock and Buena Ventura going to Eintridge and 
Academy, how long will you have to wait for the next bus? 
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Tasks were assigned values from 100 to 500 according to level of difficulty, using item 
response theory psychometric models These levels of difficult)' were detemiined "on the basis of 
the complexity of the information processing demands required for successful penomiance rather 
than by the vocabulary length or sentence length of the text alone/'^ The simplest task, requmng 
the respondent to sign his or her name on a social security card, was assigned a -. ajLje of 1 It'. A 
value of 376 was assigned to a task that involved using a page from a catalog to fill out an oriicr 
form and calculate total and itemized costs. The technical report cited indicates how these values 
were determined, but presented simply as scores they do not communicate well to the public the 
nature and level of literacy of the sampled population. 

Some other problems were found with the way the results of this sophisticated si:rv ey uere 
reponed. Results were given in three literacy domains: prose, document, and quantitative. The 
terms prose, document, and quantitative, however, do not indicate clearly the type of literacy that 
has been sampled. With relatively linle additional effort, it may be possible to present this data in 
ways that relate to the constructs of practical literacy and informational literacy that are widely 
employed in literacy research. 

On the prose scale, benchmarks were set at the 200, 275, 325, and 375 levels. On ihe 
document scaJi, they were provided for the 150, 200, 250, 300, and 350 levels. Can common 
"plausible values" be found for comparable difficulty levels on the different scales? If it were 
possible to do so, results would be easier to communicate to the lay public. Perhaps qualifiers like 
"advanced" and "proficient" could then also be applied to the benchmark levels. 

In 1993, the National Center for Education Statistics will begin to administer a survev of 
adult literacy every four years." This survey will be targeted at Americans age 16 and over. It has 



Ibid, seciion HI, p. 9. 



" The National Education Goals Panel, The National Education Goals Report, p. 207. 
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been proposed that the NAEP Young Aduli Literacy Assessmen: be expanded for this p'jr70se.- 
We support this proposal. 



Recoin.ncndations 

To gauge .progress toward the kind of educational attainmcni enNisaged by goal five, we 
recommend that the Depanment of Education employ a variety of indicators. Muiiiple niea*-ijre.> 
will be needed to convey the changes in our institutional functioning and cultural expectations that 
are to necessary to measure the progress of educational reform. The indicators and repons that we 
propose the National Center for Education Statistics (NCES) provide were chosen to enlarge the 
public's understanding about the many factors that affect change in the educational achievements of 
our population. 

There are six ways in which we would like to see tiie measurement and evaluation of 
college learning proceed. Our choice of measures follows from the preceding analysis of the best 
way to assess learning gains in the secondary schools; tiie imponance of demanding curricula in 
major fields of college learning; the role that can be played by disciplinary associations in changing 
patterns of college study; the ways in which colleges and universities are establishing learning 
outcome measures for their students; the gulf between workplace needs and the perceived 
capabilities of recent school graduates; and the need for continued monitoring of literacy-related 
skills in the adult population. 

I. Advanced Placement Exams and Courses — The Advanced Placement program successfully 
provides high school students with the opportunity to take college-level courses. We believe that 
the amount of college-level learning that goes on in the high school should be measured as part of 
the progress toward goal five. We propose that three indicators be used for this purpose. The first 

" Educational Testing Service and Wesiat, "Literacy Definition Commiilce Proposal for a National Adult 
Literacy Survey** (Submitted to Office of Educational Research and Improvement, 1991). 
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will indicate the availability of AP courses at the individual high school level. Curren:;> . les^ ih:in 
half of our high schools offer this opponunity.^' Data should be published annually by ihe NCES 
that ^ 'ill show the portion of high schools that offer one or more of these courses. The second 
indicator, which was employed in the Goals Panel Repon for goal three, tallies the number of 
students who take these course examinations, and creates a trendline with this daia.^' Curreniiy, no 
more than onc-quaner of those who take the SAT as seniors have also registered for one or more 
AP examinations.'^ The third indicator can be the success of these students on their examinaiion, 
as it is gauged by their scores. The portion of those receiving a three or higher on the common 
five-point scale should be reported. 

2. Curriculum Coherence — ^A transcript survey supplemented by interviews and course catalogs is 
a non-intrusive way of determining what types of courses are being taken by college student.'v 
across the country. We believe that an annual survey of snidents transcripts from 30-50^ 
institutions, modeled after the one conducted by Zemsky and his colleagues at the University of 
Pennsylvania, would provide valuable infonniidon on the amount of advanced study in 
undergraduate major fields. Zemsky's analysis provides data on the extent to which seniors are 
registering in demanding courses. This can be determined by an analysis of coded transcripts, with 
very little in the way of interviews or catalog review. We believe that this is a useful indicator of 
both institutional depth in college programs and student choice of demanding courses of snidy. 
The CNES should construct trendline data on depth in the undergraduate program. Current 
findings are that students in areas outside science and madiematics do not have sufficient depth in 
their programs. The effect of publishing these repons annually is likely to be significant effort by 
colleges and universities to monitor what major programs expect of graduating seniors. 

3. Direction to Undergraduate Study from the Major Disciplinary Associations — Guidance from 
disciplinary organizations can also play a significant role in defining expectations for student work 

" The CoUcge Board, AP Yearbook 1990 , p. 2. 

The National Education Goals Panel, The National Education Goals Report, p. 1 3 . 
"The College Board, Yearbook 1990 and The Ccl'.fge Board. 1991 Profile of SAT and Achievemeni Test 
Takers (New York: CoUege Entrance Examination Board. 1991). 
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in major fields of study. Those expectations can be framed in terms of what ihc vsorkplacc 
requires and post-collegiate schooling demands. Some associations. like those in mathematics and 
history, have devoted major efforts to clarifying those expectations. Others have barely begun to 
reflect on the relationship between the demands of graduate research, applied workplace activity, 
and undergraduate programs. We believe that an inventory of the guidance available rrom the 
different associations, by field, should be issued regularly. It should list those organizadons that 
have submitted recommendations, along with those that have not. Such a listing would put 
pressure on disciplinary associations that have in the past neglected to assume adequate 
responsibility for undergraduate programs. 

4. College and University Assessment Programs— Given the variety of institutions in American 
higher education, no single assessment can recognize the many kinds of learning that institutional 
programs encourage. We believe that an assessment process should encourage bpth progress 
toward high standards and faithfulness to different institutional missions. The most desirable way 
to foster both these goals is through demanding major programs that I'all for capstone senior 
experiences. These can take many fomis, but we call anention to comprehensive written and oral 
exams, theses, projects, portfolios, self-repons and a variety of demonstrations.. These programs 
can encourage students to integrate and extend knowledge in their major areas of study. 

The assessments should be institutionally generated and faculty owned, sc that faculty will 
be committed to them and they can evolve over time in response to local needs. The assessment, 
will take many foims. We propose that an inventory be created of inst^mtions with assessments in 
the major field, and that the inventory be updated annually. This inventory should not only report 
which institutions have assessment programs, but also how these programs compare to one 
anotiier in setting high standards and reflecting institutional missions. 

5. A Survey of Employers— The innovative Harris survey offers a useful method for measuring 
perception of the preparedness of high school graduates. We believe that a similar survey should 
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be designed to gauge the extent to which recent college graduates are able to meet the need- of 
today's workplace. The survey we propose would be administered on a yearly basis to a 
representative sample of employers and recent college graduates, using the sampling methods 
proposed in the original survey. Introducing additional questions with respect to job coniexta. 
types of student majors, types of employers, etc., would require a much larger surv ey sample, and 
may be unmanageable, given the varieties of collegiate education and ^ofrifoTci aciiviiy. The 
current suivey ha: the vinue of its directness and simplicity. 

We do not recommend extending the survey to parents of recent college graduates. The 
perceptions of these parents are less likely to be informed by knowledge about the capabilities of 
their children as college students than they were about them as high school, students. By contrast, 
we do recommend that educators be inter\'ie\ved. We want to not only measure cha.nges in their 
perceptions, but through publication of these reports, to focus attention on the need to raise 
expectations for learning in the colleges and universities, and indicate the gulf— which we hope 
will narrow— between what employers want and what they are likely to find among new entrants 
to the workforce. 

0. Expansion of the NAEP Young Adult Literacy Assessment— We believe that the Depanment of 
Education's proposed literacy survey of the adult population will be a valuable measure of the 
capabilities of the working-age population. We suggest tliat the survey be designed in a manner 
that permits disaggregation of various segments of the population, in particular itcent college 
graduates. This can be done through the type of oral background interview used for the NAEP 
Young Adult Literacy Assessment. Respondents, however, consistent with the goal of improving 
the quality of the college educational experience, should be asked to describe the adequacy of their 
course progiams for the demands made upon them in the workplace. The types of occupations in 
which they are employed should also be identified. 
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Conclusion 

Goal five of America 2000 calls for a transfonnation in the educational attainment of the nation's 
adult population. Specifically targeted by the goal are college graduates, who are expected bv the 
year 2000 to increasingly "demonstrate an advanced ability to think critically, communicate 
effectively, and solve problems." This paper has focussed on a number of measures that can help 
to assure that our college grav u&tes possess these capabilities. 

The gains in achievement targeted by goal five should not, in our viev^', be separated 
anificially from the other changes in attainment that are called for by the other five goals. Thus, the 
information that we propose the National Center for Education Statistics provide will not only 
enlarge the public's understanding of factors that affect change in the educational achievements of 
our college population, but also indicate the interdependence in achievement at different levels of 
schooling. 

We believe that the nation is quite far ftom achieving the levels of attainment called for by 
goal five. For this reason, we have proposed six measures that are intended to both gauge and 
promote achievement Each suggests a direction for change that we believe that our educational 
institutions should follow and a criterion by which progress in that direction may be judged. 

The proposed "basket of indicators" targets the following domains: the availability of 
Advanced Placement courses and student success in them; coherence and depth in college 
programs; contributions of disciplinary associations in guiding the structure of undergraduate 
major programs; the creation and operation of departmental assessment programs; perceptions of 
employers and educators about the woric readiness of college graduates; and levels of literacy in the 
country's adult and young adult population. 

The measures proposed here, particularly the first four, are intended to realize the ambitions 
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of goal five through changes in educational processes in our colleges and high schools. Tne last 
two measures arc nwre sensitive to learning outcomes, and offer some opportunity to gauge the 
effects of program changes at different institutions on the literacy of young people and perceptions 
of their work readiness. 

There will certainly be some negative response to the use of multiple measures that are so 
sensitive to process and fail to focus exclusively on outconws. In our view, however, outcomes 
are very context dependent If we seek better performance from college graduates, we will need to 
change the behavior of many actors: students, teachers, deparmient heads, institutional leaders, 
disciplinary associations, parents, and employers. The use of these multiple measures is intended 
to guide changes by each of these actors and communicate what has taken place to the broader 
public. No single mcasi'xo can satisfy our many needs. 
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Paper: Evaluating Progress Toward Goal Five: A Report to the National 
Center for Education Statistics 

Authors: Daniel P. Resnick and Natalie L Peterson, Carnegie-Mellon 
University 

Reviewer: Nancy Beck, Educational Testing Service 

This was a thoughtful paper that proposes expanded use of specific 
and currently available instruments or methods to collect information 
about the state of higher education. It is the authors' premise that 
organized collection and publication of information based on use of 
these instruments will lead to improvement. 

This paper reflects the strong orientation of Resnick and Peterson toward the 
importance of the major. They are up front in their belief that higher education 
takes place in the major; if you improve the quality and substance of study in the 
major fields you will improve higher education. 

If the task was to "...identify, defme, and assess a specific set of skills consistent 
with objective 5.5 - conmiunication, critical thinking, and problem solving..." then 
this paper addresses them only obliquely. Several sets of skills are discussed, but 
they are not related back to the specific skills of Goal 5, and no attempt at 
definition is made. 

What the authors do, instead, is identify six currently available measures/methods 
and discuss how expansion of each will improve higher education. The strength of 
these links varies - some have promise, others are fairly tenuous. The stated goals 
of the panel are addressed rhetorically but not substantively. 

Since Resnick and Peterson did not approach their task through the specific skills, 
applying the evaluation criteria to each of their six reconmiendations did not 
prove very satisfactory. However, using the criteria as an organizing mechanism 
was useful in reviewing the measures/methods which were presented as "...six ways 
in which we would like to see the measurement and evaluation of college learning 
proceed." (p. 21) 

1. Advanced Placement Exams and Courses 

The Advanced Placement Program is to be used as an 
indication of the amount of college-level learning going on 
in American high schools. This information, in turn, 
provides a measure of progress toward Goal 5. It is not 
made clear, however, how or why Advanced Placement 
courses or scores provides a measure of the specific 5.5 
skills, or how reporting: (1) availability of AP courses at 



individual high schools, (2) number of students who take 
these courses and exams, and, (3) success of these students 
on the AP exams will move us closer to the Goal. 

Transcript Survey 

An annual survey of transcripts from 30 - 50 institutions 
would provide a unobtrusive way of determining what 
courses students are taking. The authors see this as an 
indicator of both institutional depth in programs and of 
student choice of demanding (or not-so-demanding) 
courses of study. They further argue that The effect of 
publishing these reports annually is likely to be a 
significant effort by colleges and universities to monitor 
what major programs ex^ ct of graduating seniors." (p. 22) 

Such a survey no doubt would provide interesting and 
useful information about higher education. It is not made 
clear in this paper, however, how the particular collection 
of courses taken by a student is related to the level or 
acquisition of the skills under question. 

Direction to Undergraduate Study Provided by the M^or 
Dlsciplinaiy Associations 

The thesis here is that the disciplinary associations must 
become more active in defining - and raising - the 
expectations for student work in the major fields of study. 
The authors' strong belief that the major is where any 
meaningful education takes place is reflected in their 
arguments for this recommendation. 

Once again, this is an excellent idea with much to 
recommend it; in all likelihood it would have a positive 
influence on education. It does not, however, lend itself in 
any obvious way to the task of identifying, defining, and 
assessing the skills at issue. 

College and University Assessment Programs 

As the authors describe this activity it is another way to 
encourage demanding programs in the major. It consists of 
a published inventory - to be updated annually • of 
institutions with assessments in the major. 
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This is another recommendation that could have a positive 
influence on higher education, but it would provide only a 
very indirect measure of the skills at issue. 

5. A Survey of Employers 

This is a recommendation to adapt, for use with college 
graduates, die Harris survey that asked employers about 
their perceptions of the preparation of high school 
graduates. It would evaluate the 5 J skills where they are 
used ' in the work place. 

If the adaptation were carefully done to get at the skills in 
question, and the validity and acceptability of self-reported 
(student) and perceptual (employer) data can be 
established, this proposed measure could be potentially 
very useful. An added attraction is that it could be done at 
relatively low cost and implemented fairly quickly. 
Organizations that specialize in this kind of research 
abound. 

6. Expansion of NAEP Young Adult Literacy Assessment 

Use of this "...valuable measure of the capabilities of the 
working age population" (p. 24) is the paper's final 
recommendation. In this one, the authors do not go far 
enough. They propose only to expand the survey by asking 
respondent about "the adequacy of their course program" 
and the occupations in which they are employed. To assess 
the skills specified in Goal S, the nature of the tasks 
surveyed would have to be changed drastically. The survey 
as presently structured focuses on literacy (not surprising, 
given its title!), not on "advanced ability to think critically, 
communicate effectively, and solve problems." In addition, 
the assessment would need to be directed toward recent 
college graduates, not the general population. 

If expanded and modified to address the skills and the 
population at issue, and strengthened by further research 
and instrument development, this recommendation has 
great potential for addressing in a serious way the issues 
and concerns of the National Goals Panel. 
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Qeneral Comments 

The weakness of the Resnick/Peterson paper is that too many of their 
recommendations use instruments or techniques that are narrowly focused and 
limited in their applicability to the issue at hand. The real strength, however, is in 
reminding us that there are a number of extant instruments that, with some 
adaptation, have real potential for moving us forward. This paper offers not so 
much a cure as a way of getting a better patient histoiy. 



■4- 



32 



Comments on a Position Paper 

Evaluating Progress Toward Goal Five: A Report to 
the National Center for Education Statistics 

by Daniel P# Resnlck and Natalie Peterson 
Reviewed by: Norman Frederlksen 

This paper In unusual In Its contents, as compared vlth the other 
papers I have revleved# It suggests a number of potentially useful ways to 
Improve Instruction and to create Indicators of progress, and the Ideas came 
from commercial establishments, employer surveys, NAEP, and some colleges and 
universities, I am glad to note that the authors did not limit themselves to 
"Goal 5-Objectlve 5" because It Is vague and Is limited to a highly generalized 
concept of thinking* 

There are some topics that I wish had been Included; they have to 
do vlth performance tests .-tests that simulate problem situations as they 
appear in the real world. 

One example of a performance test is a type of test that was 
invented in the late 50s and is still being used by many large corporations, 
according to a friend who pioneered its use at ATfitT# It is an In-basket 
test .-a test that simulates tasks often required by officials in a business, a 
school, a government agency, or any other organization. It in now used 
primarily for selection and training of new employees, but it could be adapted 
to the assessment of thinking in different domains* 

The name came from the boxes found on the desks of Naval 
officers— two boxes labeled IN and OUT* The in-basket contains the 
mail' -letters, memos, directives .-and the examinee's Job is to read the mall 
and write replies* 

In order to set up a scoring procedure, the responses written by 
many subjects were sorted into a much smaller set of categories* Scoring them 
involved matching the written responses to the categories, which makes possible 
a qualitative description of perfomance* Another possibility in to get a panel 
of judges to assign numerical values to the categories, and thus obtain 
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numerical scores based on the categories 9 The Idea could easily ba adapted to 
many situations. (Sos Hemphill, J. R., Griffiths, D. E., & Frederlksen, N. 
(1962), Administrative Performance and Personality ^ Nev York: Teachers College, 
Columbia University; and Froderlkeen, N«, Jensen, 0., & Beaton E. A. (1972), 
Prediction of Orgflnlzatlonal Behavior, Nev York: Pergamon Press.) 

Another example Is a test that vould be useful In assessing and 
training higher-order thinking skills; It Is called Formulating Hypotheses. 
The test simulates a situation In vhlch a ^'researcher**, (the subject) must 
Interpret the results of his/her experiment. The researcher studies a graph or 
table shoving the results of the experiment, and reads a statement of the 
finding (vrltten by an assistant). Then the researcher vrltes hypotheses that 
might account for ths finding. 

The scoring procedure again In based on a classification of ths 
Ideas vrltten by a group of subjects, and expert Judges assign a value to each 
category. Then scores can be generated by matching hypotheses to the 
categories. 

Other tests of this kind have such titles as Evaluating Proposals, 
Solving Methodological Problems, and Measuring Constructs. Such tests 
certainly require knovledge of a domain and ability to fit their Ideas to the 
data. (Sas Frederlksen, N., & Ward, W. C. (1978), Measures for ths study of 
creativity In scientific problem solving. A pplied Psychological Measurement ^ 
2,, 1-24; and Ward, W. C, Frederlksen, N., & Carlson, S. B. (1978), Construct 
validity of f ree-respones and machlne-scorable versions of a tent of scientific 
thinking. Journal of Educational Measurement s 17, 11-29.) 
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Review of 

Evaluating Progresa Toward Qoal Five: 
A Report to the National Center for Education Statlatlcs 

by 

Daniel P. Reanlek and Natalie L Peteraon 

Review by Joa^^ Herman, UCLA/CRESST 

Draft 



The Resnick and Peterson paper presents a number of concrete 
recommendations for assessing progress toward Goal Rve, focusing 
on the objective of "measuring college student learning, particularly 
the learning of four year college graduates." The recommendations 
are well presented and solidly supported. The authors clearly 
understand the policy context and circumstances which motivate the 
national goals, goal five among them, as well as the context and 
complexities of assessment In higher education. The paper presents 
an interesting perspective on the role of assessment In promoting 
progress. 

Resnick and Peterson caste their argument and their proposed 
measures in a broader policy context and the argue that the role of 
assessment, and the indicators which comprise it, is both to monitor 
progress toward national goals and to stimulate or motivate such 
progress - as the authors state, "the indicators wilt also be 
expected, perhaps in visible ways, to mobilize public action." (p. 2) 
Indicators, in their view, should provide direction on the ways in 
which post secondary Institutions should shift to Increase the 
capabilities of their students; indicators thus have a proactive 
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function in the change process. In fact, one of the indicators 
Resnicli and Peterson propose, the "contributions of discipiinary 
associations to the structure of undergraduate major programs- 
seems to have almost solely a motivation function, i.e., to help 
assure that disciplinary associations will be active in setting 
standards for higher education, presumably thereby providing some 
grounding for future measures of goal five as well as focusing 
attention on what those standards. 

Resnicl< and Peterson reasonably argue that SCAN's skills can 
be a b^sis of measures for goal 5, but that the definition of these 
skills need to bo enriched to reflect competencies which should be 
expected of college graduates. ' This Is one basis for their 
recommendation for indicators of the coherence and depth of college 
programs - that is, college programs need to provide students with 
the opportunities to gain depth of disciplinary knowledge: the 
measure of opportunity provides a somewhat feasible and minimum 
proxy for student outcomes, as well as an important process 
indicator in support of such outcomes. In terms of more direct 
measures of these outcomes, Resnick and Peterson argue for an 
expansion of the NAEP Young Adult Literacy Assessment; While 
showing solid understanding of the existing instrument as well as 
the need to adapt It for assessment of college outcomes, the paper 
could provide more specific suggestions about the ways that 
assessment would need to be adapted and augmented - for example, 
what skills should be particularly emphasized? 

In terms of the types of items which should be emphasized, 
Resnick and Peterson clearly favor complex, performance oriented 
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measures. They argue strongly against norni-referenced measures 
and against multiple choice measures. While I would agree that 
performance and criterion-referenced measures will be important, I 
believe it would be a mistake to discard all multiple choice items; 
furthermore, while criterion-referenced measures are more helpful 
for formative purposes, norm-referenced comparisons - 
particularly in the international arena - also will continue to be 
important. 

I strongly agree with Resnici( and Peterson that a variety of 
indicators will be needed to assess Coal 5, and that those indicators 
need to consider both process and outcomes. The set they advocate - 
- advancement placement exams and courses, curriculum coherence, 
college and university assessment programs, employer and graduate 
surveys, and expansion of the NAEP Young Adult Literacy Assessment 
makes sense, although the costs for rigorous assessment in these 
areas will be more than substantial. For example, transcript studies 
to assess curriculum coherence will be very expensive, and 
something more than surface analysis may be required to assure that 
course sequences actually reflect substantive rigor. (Similarly with 
the proposed Indicator of college assessment programs. Just 
requiring a capstone experience is insufficient: such experiences 
must have rigorous standards.) Furthermore, if one of the purposes 
of such assessments is to motivate reflection and Improvement, the 
sample of colleges studied will have to be both substantial and 
annually changing; that is. one of the ways such an assessment can 
serve as a motivator is if every college feels its curriculum has a 
reasonable change to come under public scrutiny. 
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The Harris-Poll typ« approach seems a cost effective way to 
gather information from college graduates and their employers about 
student preparedness. The student survey for this and the 
recommended augmentation of the NAEP Young Adult Utetacy 
Assessment might also want to consider measures of level of effort 
•• for example. Pace's College Student Experiences Questionnaire 
(Pace, 1987) - as a proximate measure of program rigor and student 
engagement. 

Rnally. Resnick and Peterson make an excellent point about 
the importance of reporting and communication issues. If progress 
indicators are to function proactively and to shape behavior, it is 
mandatory that they be well and easily understood by those whose 
behavior is to be shaped as well as by those who are supposed to 
exert influence for change - e.g.. the media and the public. There Is 
considerable tension between the need to simplify information to 
make it understandable and the danger of compromising its integrity. 
Furthermore. R40 attention needs to be given to clear and effective 
communication to the variety of audiences who are intended to use 
the informrtlon. 
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